Massive Dirac surface states in topological insulator/magnetic insulator 
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Topological insulators are new states of matter with a bulk gap and robust gapless surface states 
protected by time-reversal symmetry. When time-reversal symmetry is broken, the surface states 
are gapped, which induces a topological response of the system to electromagnetic field-the topo- 
logical magnetoelectric effect. In this paper we study the behavior of topological surface states 
in heterostructures formed by a topological insulator and a magnetic insulator. Several magnetic 
insulators with compatible magnetic structure and relatively good lattice matching with topologi- 
cal insulators Bi2Se3, Bi2Se3, Sb2Te3 are identified, and the best candidate material is found to be 
MnSe, an anti- ferromagnetic insulator. We perform first-principles calculation in Bi2Se3/MnSe su- 
perlattices and obtain the surface state bandstructure. The magnetic exchange coupling with MnSe 
induces a gap of ~54 meV at the surface states. In addition we tune the distance between Mn ions 
and TI surface to study the distance dependence of the exchange coupling. 

PACS numbers: 73.20.-r, 85.75.-d 



Topological insulators (TI) are new states of quan- 
tum matter which has the same symmetry as the con- 
ventional insulators and semiconductors but cannot be 
adiabatically deformed to them without going through a 
phase transition. Recently, time-reversal invariant (TRI) 
TI's are theoretically predicted and experimentally real- 
ized in both two and three dimensions (2D and SD).^""^ 
A TRI TI is characterized by robust surface states and 
unique, quantized response properties, just like the quan- 
tized Hall conductance in 2D quantum Hall states. For 
3D TI the topological response is the topological mag- 
netoelectric (TME) effect^, which is a magneto-electric 
effect with magnetization M generated by electric field 
E with a quantized coefficient. The TME effect occurs 
when the surface states of TI become gapped due to time- 
reversal symmetry breaking, and is a generic property of 
3D topological insulators, which can be obtained theo- 
retically from generic models and from an effective field 
theory approach^, independently of microscopic details. 
Various consequences of the TME effect have been pro- 
posed, including Faraday/Kerr rotation of linear polar- 
ized lighti^ifr>S, the image monopolc effect^, the charge 
carried by a mangnetic monopole^'''^^,and other types of 
coupling between the charge and spin degree of freedom 
at the TI surfacei^ii^. Experimental progress has been 
made recently on the Faraday/Kerr effect in 3D TliirJZ, 
but the quantized effect predicted have not been observed 
yet. 

To realize the TME effect it is essential to introduce 
time-reversal symmetry breaking (T-breaking) at the sur- 
face of TI to make the surface insulating. There are two 
possible physical ways to open the T-breaking gap at the 
surface. The first approach is to introduce magnetic im- 
purities such as Mn or Fe to topological insulators. Both 
the Dirac-type surface states''^ and the bulk statesiS can 
mediate ferromagnetic coupling between magnetic impu- 
rities and thus induce ferromagnetic order under proper 



conditions. The surface state gap induced by doping 
magnetic impurities Mn and Fe has been observed in 
various experiments including transport, angle-resolved 
photo-emission (ARPES) and X-ray magnetic circular 
dichroism (XMCD)^"— This approach has the advan- 
tage of simple experimental setting which can be realized 
in both bulk materials and thin films. However, the sur- 
face state gap is usually small and probably non-uniform 
due to the low density and disorder effect of the impuri- 
ties. This may explain why Hall measurements on thin 
films have observed a large anomalous Hall effect but the 
longitudinal conductivity is still nonzero The second 
approach is to make a heterostructure between a topo- 
logical insulator and a magnetic insulator (MI), so that 
the surface states are gapped due to exchange coupling 
with the MI. Compared to the first approach, this ap- 
proach has the potential to achieve a stronger and more 
uniform exchange coupling and thus realize insulating 
surface states and TME effect in a higher temperature. 
The main challenge of this approach is to find the suit- 
able material for the MI, which can form a high-quality 
heterostructure with topological insulators and lead to a 
strong exchange coupling. This is the goal of the current 
work. 

In this paper we study heterostructures of the Bi2Se3 
family of TI with Mi's. We first carried an extensive ma- 
terial search and identify several materials which have 
relatively the best lattice matching with TI. The candi- 
date materials are summarized in Table U and their prop- 
erties will be discussed in more detail in the next section. 
It is important to notice that some anti- ferromagnetic in- 
sulators can also be used to introduce the surface state 
gap, although a ferromagnetic exchange coupling at the 
surface is required. Since the exchange coupling is very 
short-ranged, it will be dominated by the first layer of 
magnetic ions at the interface of TI and MI. Therefore 
if an anti-ferromagnetic insulator has magnetic moments 
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which are aligned to each other in each layer parallel 
to the surface, but staggering in the perpendicular direc- 
tion, it provides the same form of ferromagnetic exchange 
coupling as a ferromagnetic insulator does. In fact, the 
best candidate material found in our search is an anti- 
fcrromagnctic insulator, MnSe, which is a better candi- 
date than ferromagnetic insulators such as EuS since the 
latter has magnetism from /-electrons and has a weaker 
exchange coupling with the p-electrons in TI compared 
with the d-electrons in MnSe. We present ab initio cal- 
culation in MnSe/Bi2Se3 superlattices, from which we 
obtain the surface state gap and also describe it in a 
surface state effective model. In addition we tune the 
distance between Mn ions and TI surface to study the 
distance dependence of the exchange coupling. Then we 
conclude by discussing the band-bending effect caused by 
the interface charge at the TI/MI junction, which is the 
main experimental challenge that needs to be addressed 
in future works. 

Criteria and candidate materials 

The topological insulators Bi2Se3, Bi2Te3, and Sb2Te3 
have layered structure, and the 2-dimensional lattice 
within each layer has triangular symmetry. The 2D lat- 
tice constants of Bi2Se3, Bi2Te3, and Sb2Te3 are 4.1355 
A3i, 4.395 and 4.264 A2£, respectively. We look for 
candidate MI materials with 2D crystal plane of compat- 
ible symmetry to TI layers, ie. hexagonal lattice. The 
other criteria for the candidate MI materials include sim- 
ilar lattice constant, and ferromagnetic moments in the 
2D hexagonal interface atomic plane. 

A list of candidate MI materials is shown in table ID 
EuO^I, EuS^, EuSe^, and MnSe^O have the cubic rock- 
salt structure, of which the atoms in the (111) plane form 
triangular lattice, with compatible lattice constant to the 
TIs. Both EuO and EuS are ferromagnetic insulators, 
which meets the requirement of correct magnetic config- 
uration, however the lattice constant of EuO is too small 
for good lattice matching with the common 3D topo- 
logical insulators. EuSe has a complex magnetic phase 
diagram, and it also becomes FM under suitable pressure 
range. Magnetism in EuO, EuS and EuSe originates from 
the half-filled 4/ orbitals of the Eu+^ ions. MnSe has the 
type-II (G-type) anti-ferromagnetic structure, of which 
the magnetic moments in each (111) atomic planes are 
ferromagnetic. MnTe^ is chemically similar to MnSe, 
although it has a hexagonal lattice. Mn moments in 
MnTe also have an AFM configuration, formed of al- 
ternating hexagonal (0001) FM planes. Magnetism in 
MnSe and MnTe comes from the high-spin orbitals of 
the Mn+^ ions. RbMnClj?^ is also in hexagonal struc- 
ture, with the Mn sites forming alternating (0001) FM 
planes. Although its hexagonal plane has a much larger 
lattice constant, it matches the \/3 x \/3 reconstruction of 
the in-plane lattice of Bi2Se3. Thus RbMnCls is a poten- 
tial candidate material, although the magnetic coupling 
would be weaker because only 1 /3 of the atomic sites of 
the TIs will be in contact with the Mn sites. 

In the candidate materials listed in Table HI we 




FIG. 1: (color online), (a) Schematic picture of a MnSe film 
deposited on the surface of Bi2Se3. (b) Crystal structure of 
the supercell structure in the first-principles calculations. 



carried ab initio calculations to several of them in- 
cluding MnSe/Bi2Se3, MnTe/Bi2Se3, MnTe/SbaTeg, 
EuS/Bi2Se3 and EuSe/Bi2Se3. Among these heterostruc- 
tures, we find MnSe/Bi2Se3 to be the best one, with rela- 
tively strong exchange-coupling and simple surface state 
band structure. In the following we will focus on the 
results of MnSe/Bi2Se3 heterostructure, and present the 
results on MnTe/Sb2Te3 and MnTe/Bi2Sc3 in the ap- 
pendix as a comparison. 
Bi2Se3/MnSe interface 

We study the Bi2Se3/MnSe interface by constructing a 
superlattice composed of Bi2Se3 slab and MnSe slab. The 
supercells employed in the first-principles calculations are 
required to have inversion symmetry, and the Mn atoms 
at the top and bottom surfaces of Bi2Se3 slab have par- 
allel spin orientation. The thickness of the Bi2Se3 slab is 
chosen to be 4 QLs, the MnSe slab is determined to have 
a thickness of 6n+l (7, 13, 19, . . .) layers, in order to meet 
the requirements of inversion symmetry and parallel spin 
orientations. To simulate the physical system of a MnSe 
film deposited on top of a bulk Bi2Se3, the in-plane lat- 
tice constant of the MnSe is fixed to that of Bi2Se3, and 
the out-of-plane lattice constant is determined by energy 
optimization. The 4-7 supercell structure (composed of 4 
QLs of Bi2Se3 and 7 layers of MnSe) is shown in fig.[TJb). 

First-principles density functional theory (DFT) cal- 
culations are performed to optimize the supercell struc- 
ture and to obtain the electronic band structure. We 
used the Hohcnberg-Kohn density-functional theory with 
the generalized-gradient approximation (GGA)'^'^ and the 
projector augmented-wave method as implemented in 
VASB^i-^.The Mn 3d orbitals are treated with the GGA 
plus Hubbard U (GGA+U) method^i^, and typical val- 
ues of correlation parameters are used^i^: U — 5.0 eV, 
and J = 1.0 eV. 

If not noted otherwise (as when tuning the separation 
distance between Bi2Se3 and MnSe slabs), the atomic 
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TABLE I: A list of candidate magnetic insulators and their properties. 



Materials 


btructurc 


Lattice matching 


Magnetic phase 


Comments 


EuO 


rocksalt, 


5.145/\/2 


FM 


4/ electrons 




a = 5.145 A 


= 3.64 A 




weak coupling 


EuS 


rocksalt, 


5.98/^ 


FM 


4/ electrons 




a — 5.98 A 


= 4.23 A 




weak coupling 


EuSe 


rocksalt, 


6.192/^ 


complex, 


4/ electrons 




^ CI no X 
a = 0.192 A 


= 4.3o A 


r M under r 


weak coupling 


MnSe 


rocksalt , 


5.464/^2 


type-11 AFM, 


3d electrons 




a = 5.464 A 


= 3.86 A 


FM (111) planes 


stronger coupling 


MnTe 


hexagonal. 


4.1497 A 


AFM, 


3d electrons 




a = 4.1497 A 




FM (0001) planes 


stronger coupling 


RbMnCIa 


hexagonal. 


7.16/V3 


AFM, 


only 1/3 of 




a = 7.16 A 


= 4.13 A 


FM (0001) planes 


lattice points 



coordinates of the superlattice are optimized as in the 
following. The middle part of the Bi2Se3 slab is fixed to 
the experimental structure, while the atomic coordinates 
of the whole MnSe slab together with the first Bi and 
Se atomic planes of Bi2Se3 at the interface are optimized 
according to the forces from first-principles calculations. 
Bandstructure.- We first investigate the 4-13 
Bi2Se3/MnSe superlattice, with Mn spins along 
[001] direction. The bandstructure from first- principles 
calculations is shown in fig. [51 By projecting the bands 
to the Bi and Se atoms of the top and bottom surfaces, 
shown in fig. HJa), we identify the Dirac cone feature 
located about 0.4 eV below the bulk gap of Bi2Se3 as 
surface states of Bi2Se3. Further analysis on the spin 
directions of the states around F point confirms it to be 
the topological surface state. A small gap appears at the 
Dirac point, and both the lower and upper Dirac cones 



show spin related energy splitting in the vicinity of F 
point, indicating the magnetic interaction with MnSe. 

The band gap of MnSe is much larger than that of 
Bi2Se3. In order to obtain a clearer picture of the elec- 
tronic states of the two materials forming the superlat- 
tice, we project the bands separately to the Bi2Se3 slab 
and the MnSe slab in the superlattice, shown in fig.[51Jb). 
The bulk gap of Bi2Se3, as well as the Dirac cone states, 
are located in the band gap of MnSe, which makes it 
possible to realize a fully gapped system and observe the 
TME effect. 

Effective model fitting. - To gain better understanding of 
the exchange coupling experienced by the surface states, 
we introduce an effective Hamiltonian i?eff for the Dirac 
fermion surface states of the topological insulator thin 
film coupled with the exchange field^ 



J 



cff ■ 



Dk^I 



O'ykx) + M ■ (T 

tl 



tl 

'hvF{(Jxky - (Tykx) + M ■ a- 



(1) 



r 



Here ai {i = x, y, z) are the Pauli matrices, and / the 
identity matrix. M and t correspond to the effective 
magnetic field acting on the Dirac fermion surface states 
and the inter-edge interaction between the two surfaces. 
The other parameters in the model are: D, the quadratic 
term, and vp the Fermi velocity. 

The eigen-energies of the model Hamiltonian can be 
solved analytically. 



E = Dk^±\jk:^ + K'P + t^± 2^MH^ + (M^ky - Myk^)'- 

(2) 

here we set hvp = 1- We note that at F point (fc = 0), 



the four eigen-energies are E = ±(M ± t). In the case of 
strong magnetic coupling and weak inter-edge interaction 
where M > t, the zone-center eigen-energies (from low 
energy to high energy) and the corresponding spin orien- 
tations are: —M — t (spin down), —AI + t (spin down), 
AI — t (spin up), and M + t (spin up). In the opposite 
case where AI < t, the zone-center eigen-energies and 
spin orientations are: —M — t (spin down), AI — t (spin 
up), —AI + t (spin down), and AI + t (spin up). In both 
cases, an energy gap A forms at the original Dirac point, 
A = 2\M-t\. 

The Dirac fermion bandstructure obtained from first- 
principles calculations is fitted to the model solution in 
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FIG. 2: (color online). Bandstmcture of the Bi2Se3/MnTe 
superlattice. The two panels show the projection of the wave- 
functions to the Bi and Se atoms at the top and bottom sur- 
faces (a), and that to the two materials Bi2Se3 and MnSe (b). 
The calculation is done for the superlattice structure com- 
posed of 4 quintuple layers of Bi2Se3 and 13 layers of MnSe, 
with Mn spins along [001] direction. 

equation [2] For the superlattice structure composed 
of 4 quintuple layers of Bi2Se3 and 13 layers of MnSe 
with Mn spins along [001] direction, the calculated Dirac 
fermion surface states around F point for crystal momen- 
tum along kx (F-M) direction are fitted satisfactorily to 
the model solution, as shown in figure [3l^a). The fitting 
parameters are: AI = 28.2 meV, t = 17.6 mcV, D = 
9.8 eVA^, and vp = 2.66x10^^ m/s. Fitting the model 
solution to the calculated band dispersion along ky (F- 
K) direction results in very similar parameters, D = 9.9 
eVA^, and vp = 2.70x10^ m/s. 

Distance dependence of parameters t and M To un- 
derstand the behavior of large physical system, we in- 
vestigate the dependence of the effective magnetic field 
M and inter-edge interaction t on the thickness of the 
MnSe slab in the superlattice structure. First-principles 
band structure calculations of the 4-7 (composed of 4 
QLs of BizSes and 7 layers of MnSe), 4-13, and 4-19 
Bi2Se3/MnSe superlattices were fitted to the model so- 
lution using equation [2] The fitting parameters M and 
t are plotted as a function of the MnSe layer number, 
as shown in fig. [3l^b). The effective magnetic field M 
does not vary strongly as the number of MnSe layers in- 
creases, consistent with the fact that it is mostly due 
to the magnetic exchange coupling between the MnSe 




MnSe layer number MnSe-Bi^Se^ separalion d (A) 



FIG. 3: (color online), (a) The calculated band structure 
(black circle, the same data as Fig. [2]) is fitted by the ef- 
fective model lU (blue curves), (b) The dependence of the 
inter-edge interaction t and the effective exchange field M 
on the thickness of the MnSe slab, obtained from the fitting, 
(c) A schematic picture of the material structure used in the 
calculations shown in the next panel, with the distance d be- 
tween MnSe and Bi2Se3 tuned (see text), (d) Dependence of 
M on the separation d between the Bi2Se3 and MnSe slabs. 




Energy (eV) 



FIG. 4: (color online), (a) Illustration of the charge states 
at the Bi2Se3/MnSe interface, and a schematic picture of 
band bending, where the (red) dot at the interface represents 
the surface state Dirac point, (b) The calculated projected 
density-of-states of Bi2Se3 (black) and MnSe (red) for lat- 
tice sites across the surface, showing the band bending at the 
interface. Here each curve for MnSe represents one MnSe bi- 
layer, and each curve for Bi2Se3 represents one BiSe bilayer 
in either the top or bottom half of the quintuple layers. 



and the surface states of Bi2Se3 at the interface. In con- 
trast, the inter-edge interaction t decreases rapidly as the 
number of MnSe layers increases. This indicates that the 
inter-edge interaction in smaller superlattices are medi- 
ated mostly through the MnSe slab, instead of within the 
Bi2Se3 layers. The strong effective magnetic field (M ^ 
27 meV) at this interface results in a large gap open- 
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ing (2M ^ 54 meV) in realistic experimental setup with 
thicker MnSe slab, which holds promise for magnetic ma- 
nipulation of TI surface states at room temperature. 

In our calculation a perfect interface between MnSe 
and Bi2Se3 is assumed. In reality, the interface will prob- 
ably contain impurities and defects, so that the coupling 
between Mn and Se across the surface is weaker. To 
see how the magnetic exchange coupling is affected by 
the distance between Mn and Se atoms, we now keep 
fixed the atomic coordinates within the Bi2Se3 and MnSe 
slabs, but varying the separation d between the Bi2Se3 
and MnSe slabs, as shown in fig. ^c). As the effective 
magnetic field M is mostly due to the exchange coupling 
between MnSe and Bi2Se3 across the interface, we expect 
a strong dependence of M on the separation distance 
d. For the 4-13 Bi2Se3/MnSe superlattice, the effective 
magnetic field M is plotted in fig. ^d) , showing a strong 
dependence of M as a function of the separation d be- 
tween Bi2Se3 and MnSe. This result indicates that the 
magnetic exchange coupling at the TI-MI interface could 
be tuned by applying external pressure. 
Surface charge and band bending 

Although the Dirac cone is in the bulk gap of MnSe, 
from the band structure shown in Fig. [2] we see some 
additional hole-type bands coexisting with the massive 
Dirac cone. These bands can be understood as a conse- 
quence of the band bending occuring at the heterostruc- 
ture. Band bending at a hetero junction is common. In 
MnSe, Mn^+ and Se^~ ions carry positive and negative 
charge, respectively. Consequently, the Mn terminated 
surface considered here carries a finite positive charge 
density. Since the Bi2Se3 is a covalent material and has 
no surface charge, the net charge of the hetero junction 



between these two materials is positive, leading to a trap 
for the surface electrons. Therefore the bands bend down 
around the interface, as is illutrated in Fig. EJa). leading 
to the additional surface state bands. 

We studied the band bending effect by calculating 
the electronic density-of-states projected to individual 
atomic planes. The projected density-of-states (PDOS) 
of the Mn-Se atomic planes are shown in red curves in 
fig. ID^b), while the PDOS for the top and bottom Bi- 
Se atomic planes in the Bi2Se3 slab are shown in black 
curves. It is clear that very little band bending exists in 
MnSe, while there is significant shift between the states 
of the first and second QLs in Bi2Se3. The electronic 
states in the first QL of Bi2Se3 is shifted down in energy 
compared to the second QL, consistent with the direction 
of the interface electric dipole moment shown in fig. HJa) . 

The additional surface states lead to difficulty in open- 
ing a full gap at the surface, the solution of which is the 
main open question we leave for future theoretical and 
experimental works. Possible solutions include screening 
the surface charge by a top gate, or by proper chemical 
doping at the junction. 
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Appendix A: In-plane magnetization 

As a comparison, we also calculated the bandstructure of Bi2Se3/MnSe superlattice with Mn spins along the in- 
plane [100] direction. The bandstructure along ky (F-K) direction with projection of the bands to the Bi and Se 
atoms of the top and bottom surfaces is shown in fig.El^a). Similar to the case of [001] spin orietation shown in Fig. El 
the Dirac-cone states are located about 0.4 eV below the bulk gap of Bi2Se3. One prominent feature is that the 
Dirac-cone states of the top and bottom surfaces are shifted to opposite directions along ky axis. For a Fermi energy 
above the Dirac cone, the Fermi surfaces from the top and bottom surfaces of Bi2Se3 will shift to opposite directions 
in the momentum space. Fig. [SJb) illustrates the opposite shiftof the two Fermi surfaces along ky direction, and the 
corresponding spin directions. 

By fitting the effective model ([1]) with an in-plane exchange field, we have extracted the exchange coupling M and 
inter-edge interaction t. For the 4-13 superlattice structure with in-plane magnetization, we obtain M = 9 meV, 
and t = 19 meV. The value of t mostly depends on the thickness of MnSe and Bi2Se3 slabs, so t in the case of 
in-plane magnetization is similar to the value for perpendicular magnetization. However the exchange coupling is 
quite anisotropic, and it is much smaller in the case of in-plane magnetization compared to the perpendicular one. 



Appendix B: Sb2Te3/MnTe interface 

Other materials in our study include the Sb2Te3/MnTe heterostructure. The calculated bandstructure of the 4-13 
superlattice with projection of the bands to the Sb and Te atoms at the top and bottom surfaces is shown in fig.[6ja). 
In contrast to the Bi2Se3/MnSe heterostructure, there is complicated hybridization of bulk and surface states in the 
Sb2Te3/MnTe heterostructure, and the Dirac-cone feature can not be easily identified. 

We project the bands separately to the Sb2Te3 slab and the MnTe slab in the superlattice, as shown in fig. [HJb). 
Although the bulk band gap of MnTe at F point is much larger than Sb2Te3, the bulk band gap of Sb2Te3 is located 
close in energy to the top of valence band of MnTe, and the Dirac surface states overlap in energy with the valence 
band of MnTe. Therefore it is less likely to realize a fully gapped system in Sb2Te3/MnTe heterostructure. 



Appendix C: Bi2Se3/MnTe interface 



We have also studied the Bi2Se3/MnTe heterostructure with supercell composed of 4 QLs of Bi2Se3 and 7 layers 
of MnTe. The calculated bandstructure with projection of the bands to the Bi and Se atoms at the top and bottom 
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surfaces is shown in fig. [T) Similar to the Sb2Te3/MnTe heterostructure, the surface states and bulk states in the 
Bi2Se3/MnTe heterostructure are also stronly hybridized, and the Dirac-cone feature can not be easily identified. 




FIG. 5: (Color online) (a) Bandstructure of the Bi2Se3/MnSe superlattice along ky (F-K) direction. The red and blue symbols 
show the projection of the wavefunctions to the Bi and Se atoms at the top and bottom surfaces, (b) a schematic showing the 
Fermi surfaces of n-doped surface states at the two surfaces of Bi2Se3. The Fermi energy is taken at the dashed line in (a). The 
calculation is performed for superlattice structure composed of 4 QLs of Bi2Se3 and 13 layers of MnSe, with Mn spins along 
[100] direction. 




FIG. 6: (Color online) Bandstructure of the Sb2Te3/MnTe superlattice. The two panels show the projection of the wavefunctions 
to the Sb and Te atoms at the top (red symbols) and bottom (blue symbols) surfaces (a), and that to the two materials Sb2Te3 
and MnTe (b). The calculation is done for the superlattice structure composed of 4 QLs of Sb2Te3 and 13 layers of MnTe, with 
Mn spins along [001] direction. 

I 
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FIG. 7: (Color online) Bandstructure of the Bi2Se3/MnTe superlattice, showing the projection of the wavefunctions to the 
Bi and Se atoms at the top (red symbols) and bottom (blue symbols) surfaces. The calculation is done for the superlattice 
structure composed of 4 QLs of Bi2Se3 and 7 layers of MnTe, with Mn spins along [001] direction. 



